Enhancing Data Integrity in Online Health Surveys Through Multilayered Security Measures: Cross-Sectional Study

Introduction

Online health surveys have become essential tools for public health and clinical research, yet they are increasingly vulnerable to sophisticated fraudulent responses, impacting data integrity and quality. Fraudulent responses are submissions that contain false, fabricated, or misleading information provided to misrepresent participation eligibility and that may include automated entries from bots or repeated entries by the same actor [1]. Pinzón et al [2] documented a dramatic erosion of usable data over time, underscoring how AI-enabled fraud and incentive-seeking behavior can overwhelm open links. Similar issues have been reported in nursing and medical research, where 70% to 94% or more of survey responses were identified as fraudulent or automated through bots [3-6]. These findings suggest that traditional safeguards, such as CAPTCHA, are insufficient for data integrity [3,7]. This study’s purpose is to describe the results of implementing a multilayered security approach that integrates automated and human-review mechanisms in a nationwide survey of health care professionals.

Methods

Overview

We conducted a cross-sectional online survey of health care professionals across the United States between November 2024 and May 2025. Clinicians aged 18 years or older practicing in the United States who have worked with patients with limited English proficiency were eligible to participate. The survey examined pain-related communication challenges with Spanish-speaking patients with limited English proficiency.

Recruitment occurred through medical professional organization newsletters and websites, health care–focused Facebook and WhatsApp (Meta Platforms Inc) groups, and the authors’ professional Facebook and Instagram (Meta Platforms Inc) pages. No paid advertising was used. We used a 2-tiered process, where posts included a direct link or QR code to the screening survey and then eligible respondents received an individualized link to a Qualtrics (Qualtrics LLC) survey.

Our multilayered security approach combines automated fraud indicators (Table 1) with a human verification process. Some indicator thresholds were adapted from prior literature based on patterns observed in our previous fraud detection work and in the current dataset. Using R software (version 4.5.1; R Foundation for Statistical Computing), we developed a fraud detection workflow that followed these steps: (1) entries were automatically disqualified based on eligibility criteria (eg, non–health care providers) and definitive fraud indicators (eg, non-US submissions), (2) remaining submissions underwent manual review where entries with more than 2 indicators received brief verification of fraudulent status, and (3) entries with 1 or 2 indicators underwent human manual verification, including confirming provider credentials using National Provider Identifier records or state professional licensing board databases. No submission was classified as fraudulent based on a single nondefinitive indicator.

Ethical Considerations

This study was approved by the University of Wisconsin-Madison’s institutional review board (ID: 2024‐1314). Informed consent was displayed in the introduction of the survey, and all participants had to agree to participate to continue. Survey responses were stored separately from contact information to ensure privacy and confidentiality. Participants received an electronic gift card of US $15 for completing the survey.

Results

Our multilayered security approach identified 5846 of 5886 entries (99.32%) as potentially fraudulent, while only 40 were verified to be legitimate. The automated screening phase disqualified 2895 (49.18%) submissions. The remaining 2991 (50.82%) submissions underwent manual review. Of these, 2224 (74.35%) contained 2 or more indicators of fraud and required only brief verification, while 767 (25.64%) required more extensive examination. Table 2 shows that the most frequent fraud signal was the temporal regular waves pattern (n=4447, 75.55%), followed by a high fraud score (n=1614, 27.44%) and duplicate internet protocol (n=1614, 24.88%). Combined attacks were also notable, with the regular wave  + suspicious email pattern being present in 418 (7.1%) responses (Table 2).

Discussion

Principal Findings

This multilayered security approach proved highly effective at identifying submissions with multiple indicators of fraud and distinguishing them from submissions from verified health care professionals through credential verification and manual review. Among individual signals of fraud, the most discriminative was the temporal regular wave pattern. By contrast, the honeypot captured no entries, and submissions with no contact information were rare (1.4%) and nonspecific. This suggests that contemporary threats originating from sophisticated automated systems are capable of avoiding traditional bot-detection mechanisms.

No single indicator was sufficient for adjudication because legitimate circumstances could generate these signals. For example, regular wave patterns may result from social media distribution, duplicate IP addresses may occur when using shared networks, and character-based indicators (eg, a low vowel/consonant ratio) may reflect legitimate naming conventions (eg, Schmidt). Thus, human review is essential to minimize false positives when scaling verification.

Our findings confirm that no single indicator is sufficient; detection improves when multiple indicators are combined [2,8]. Guy et al [10] reported similar results in 2 online studies of marginalized populations, showing that reliance on a single fraud detection approach would have resulted in the majority of bots or fraudulent responses remaining undetected. In our data, even the strongest single signal would have missed nearly one-quarter of fraud if used alone. Multilayered defenses enabled the accurate removal of 16% to 88% of fraudulent entries depending on context; where permissible, personal identifiers would further improve yield [3,7,10,11].

Several limitations should be acknowledged. First, we cannot formally estimate sensitivity, specificity, or overall classification accuracy, including naming traditions, because there is no gold standard to determine each submission’s true status. Second, all recruitment channels used the same screening survey; therefore, we could not compare fraud rates across channels.

As online survey fraud continues to evolve, researchers must remain vigilant and adaptive. The extreme fraud rate documented here may reflect a new reality for online studies, particularly those offering monetary incentives. While resource intensive, our results demonstrate that comprehensive fraud detection can successfully preserve data integrity even in heavily targeted surveys. Our findings also highlight the importance of recruitment strategies. Public social media posts through professional organizations and health care–focused groups may increase survey visibility and simultaneously increase exposure to fraud. Future studies should evaluate whether recruitment through professional organizations’ members-only platforms reduces fraud while maintaining adequate reach.

Conclusions

This study demonstrates that safeguarding online health survey data requires a comprehensive multitiered framework. As technology evolves, adaptive approaches will be essential to maintain the validity, credibility, and reproducibility of web-based research. Furthermore, we provide practical insights into recruiting health care professionals through professional organizations and public social media channels. Investigators should anticipate recruitment inefficiencies, and they may need to cast a wider recruitment net than anticipated and allocate substantial resources to response verification.